Papers with encoder representations
Lacuna Reconstruction: Self-Supervised Pre-Training for Low-Resource Historical Document Transcription (2022.findings-naacl)
Copied to clipboard
| Challenge: | Document transcription models are limited by extremely varied style and content across domains. |
| Approach: | They propose a self-supervised approach for learning rich visual representations for both handwritten and printed historical document transcription using a heterogeneous set of handwritten Islamicate manuscript images and early modern English printed documents. |
| Outcome: | The proposed model improves on a supervised model with as few as 30 line image transcriptions on two languages with a single line of image training. |
Cross-Domain Generalization of Neural Constituency Parsers (P19-1)
Copied to clipboard
| Challenge: | Neural parsers perform well on in-domain benchmarks, but their performance degrades in well-understood ways. |
| Approach: | They analyze generalization on English and Chinese corpora to see if they can generalize to other domains. |
| Outcome: | The proposed neural parsers perform better on in-domain benchmarks than on out-of-domain corpora. |
Hyperdecoders: Instance-specific decoders for multi-task NLP (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work in NLP has examined the performance of large pretrained transformer-based models in multi-task settings, where a single model is evaluated on multiple tasks simultaneously. |
| Approach: | They propose a method for multi-tasking using a hypernetwork conditioned on the output of an encoder to generate a unique decoder adaptation for every input instance. |
| Outcome: | The proposed method outperforms previous methods for efficient multi-task fine-tuning and maps from encoder representations to output labels. |
Sparse and Decorrelated Representations for Stable Zero-shot NMT (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Using a single encoder and decoder for all directions is a popular scheme for multilingual NMT. |
| Approach: | They propose a scheme that uses a single encoder and decoder for all directions . they show that enforcing sparsity and decorrelation on encoder intermediate representations mitigates this problem . |
| Outcome: | The proposed model degenerates when decoding non-English texts into English regardless of the target specifier token. |
Improving Multilingual Neural Machine Translation by Utilizing Semantic and Linguistic Features (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models do not differentiate between semantic and linguistic features, resulting in the entanglement of knowledge and linguistics within the model. |
| Approach: | They propose to exploit both semantic and linguistic features to enhance multilingual translation by disentangling encoder representations and integrating low-level linguistic encoders. |
| Outcome: | The proposed model improves zero-shot translation while maintaining performance in supervised translation on multilingual datasets. |
Effectively pretraining a speech translation decoder with Machine Translation data (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to improve the performance of AST systems are based on pretraining the encoder parameters using an ASR model, but using a pretrained MT decoder is not beneficial or improves the results. |
| Approach: | They propose to use an adversarial regularizer to bring the encoder representations of the ASR and NMT tasks closer even though they are in different modalities. |
| Outcome: | The proposed model can be pre-trained using the Automatic Speech Recognition (ASR) task even in different languages and improves in low resource settings. |
The Medium Is Not the Message: Deconfounding Document Embeddings via Linear Concept Erasure (2025.emnlp-main)
Copied to clipboard
| Challenge: | Embedding-based similarity metrics can be influenced by content dimensions and spurious attributes like the text’s source or language. |
| Approach: | They propose a debiasing algorithm that removes observed confounders from encoder representations and removes them from the encoder. |
| Outcome: | The proposed method improves on out-of-distribution benchmarks and on benchmarks, but performance is not affected. |